Vectorize

Querying Overview

A summary of the Cloudflare Vectorize database can be found at https://developers.cloudflare.com/vectorize/reference/what-is-a-vector-database/

The Qarbine Administrator is responsible for setting various parameters such as the interaction tokens for querying and obtaining vector embeddings.

At the bottom of this document are some Query by Example notes to be aware of as well.

One way to query Vectorize from Qarbine uses a JSON object with the following field options.

Field	Description
index	The name of the index being queried.
vector	The embedding vector for similarity querying. The dimensions of the vector must match that of the index’s creation definition.
nearTextnearTextModel	These 2 fields are used in combination. When they are specified the vector value is obtained dynamically based on these 2 values.
topK	At most how many rows to return. Vectorize supports an upper limit of 100 for the topK value. However, for a query operation with returnValues set to true or returnMetadata set to all, topK would be limited to a maximum value of 20. For details seehttps://developers.cloudflare.com/vectorize/reference/client-api/#topk
returnValues	A boolean flag to return the row vector as well.
returnMetadata	A value indicating what row metadata to return. Recognized values are ‘all’, ‘none’, and ‘indexed’. The default is ‘none’. For details see https://developers.cloudflare.com/vectorize/reference/client-api/#returnmetadata
filter	The filter criteria. See the section below for more details.
minimumScore	See the section below for details.
ids	The explicit list of vector identifiers to retrieve.

Distance metrics determine how close 2 objects are to one another. There are several algorithms supported including cosine, Euclidean and dot product. For more details see
https://developers.cloudflare.com/vectorize/best-practices/create-indexes/#distance-metrics

Sample Query Specification

Here is a sample query specification

{
nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',
index:"product-index",
topK: 5,
}

The result is shown below.

  

These 2 parameters are closely tied together

nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',

They indicate to obtain the embedding value for the test ‘mouse’ using the Cloudflare model with the identifier of “@cf/baai/bge-small-en-v1.5”. That value is then placed into the “vector” field for the request sent to Vectorize.

Adjusting the specification with the returnMetadata Vectorize option

{
nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',
index: "product-index",
topK: 5,
returnMetadata: 'indexed'
}

results in

  

Selecting the first row shows its details

  

Adjusting the specification with a Qarbine pragma option as the first line

#pragma pullFieldsUp metadata
{
nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',
index: "product-index",
topK: 5,
returnMetadata: 'indexed'
}

results in

  

Notice the fields that were previously within the metadata document have been pulled up to the main object level. This makes defining analysis templates a bit easier.

Adjusting the specification to have

returnMetadata: 'all',

results in

  

Filtering

Overview

Vectorize filters are similar to SQL “where” clauses. Any filter is applied before the similarity search, so only rows whose metadata matches the filter are considered for the vectors query. There must be a metadata index for the property you want to filter on. Filtering supports string, number, and boolean types. Filters can be applied directly in your query, and you can combine multiple filters using implicit logical AND.

Operations

Cloudflare Vectorize supports a range of filter operations for querying vectors based on their metadata. These operations allow you to precisely limit which vectors are returned in a similarity search or other query. Supported metadata filter operations include those listed below.

Operator	Description	Operator	Description
$eq	Equal to	$lt	Less than
$ne	Not equal to	$lte	Less than or equal to
$in	Value is in a given array	$gte	Greater than ror equal to
$nin	Value is not in a given array	$gt	Greater than

Filtering is case-sensitive and requires an exact match, including spaces and capitalization. "Alice Smith" must match exactly what was stored in the vector's metadata. Also, only the first 64 bytes of a string metadata field are indexed and filterable.

For more information see
https://developers.cloudflare.com/vectorize/reference/metadata-filtering/

Sample Filtering Query

The specification bellow includes a Vectorize filter

#pragma pullFieldsUp metadata
{
nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',
index: "product-index",
topK: 5,
returnMetadata: "all",
filter: {price : {$gte: 25} } 
}

The results are shown below.

  

Specifying a Minimum Sscore

Cloudflare Vectorize does not support setting a minimum score threshold directly in the query API. However, Qarbine does let you specify one in its query specification as shown below.

{
minimumScore: 0.75
…
}

This applying of the minimum score criteria is done just after the Vectorize answer set is retrieved and before any further Qarbine pragmas or other processing occurs.

Sorting the Results

By default Vectorize returns the rows in highest to lowest score order. Here is a technique to use an alternative sorting. The order of the pragma lines is important.

#pragma pullFieldsUp metadata
#pragma sortResultBy price desc
{
nearText: 'mouse',
nearTextModel: '@cf/baai/bge-small-en-v1.5',
index: "product-index",
topK: 5,
returnMetadata: "all",
filter: {price : {$gte: 25} } 
}

The result is shown below.

  

Retrieving Specific Vectors

Define a list of identifiers in the “ids” field to retrieve specific vectors. Here is an example

#pragma pullFieldsUp metadata
{
index: "product-index",
ids: [ '1', '3'],
returnMetadata: "all",
// The default is for Vectorize to return the values as well.
// We will only return them if returnValues is explicitly true.
//returnValues: true
}

A sample result is shown below.

  

Note that the identifiers are strings. If the 2nd identifier argument was just the number 3 for example then the following error would be returned from Vectorize.

{"error":"Request failed with status code 400. Code 40026. Failed to deserialize the JSON body into the target type: ids[1]: invalid type: integer `3`, expected a string at line 1 column 13\n"}

Other Vector Filtering

The nearText/nearTextModel approach above uses Cloudflare Workers to obtain the embedding. An alterative is to use Qarbine AI Assistants and macro placeholders

vector: [! embedding('mouse', 'myCloudflareBgeAssistant') !]

Here the Qarbine administrator has defined an Qarbine AI assistant with the alias 'myCloudflareBgeAssistant'. That setting’s entry references an LLM which corresponds to the one used to set the fields on the index’s rows. For more details see the Qarbine “AI Assistant” documentation.

Qarbine SQL Oriented Interaction

Overview

Qarbine provides a convenient option to use SQL syntax rather than the JSON structure described above. This is especially useful for specifying filter criteria. Below are the SQL clauses and corresponding commentary.

Clause	Comments
SELECT	The id and score value are always returned. Specify ‘*’ to return ‘all’ metadata properties. Specify ‘indexed’ to return only the indexed properties. Include ‘vector’ to return the embedding vector values. This keyword can be combined with the options above.
FROM <index>	Specify the index being searched. If there are dashes in the name then quote the name.
WHERE <criteria>	Specify the filter criteria. The operations allowed correspond to the standard Vectorize operators mentioned above. The following uses the logical ‘vector’ column reference to effectively set the embedding criteria. vector = [numbers] There are also several support functions that can be used. nearText(someText, AI_Assistant, modelName) Use a null AI_Assistant argument when directly interacting with Cloudflare’s integrated embedding service. nearVector(numbers)
ORDER BY	Vectorize returns the row matches in best score first order. If you would like a different order then review the Qarbine pragmas runPostQuery and sortResultBy.
LIMIT howMany	This clause corresponds to the topK argument. Vectorize does not like a howMany value greater than 20 when returnMetadata and returnValues are set to true. For details see https://developers.cloudflare.com/vectorize/reference/client-api/#topk

Sample Query

Below is a sample SQL oriented query.

select *
from "product-index" 
where nearText('mouse', null, '@cf/baai/bge-small-en-v1.5')
and price >= 25
limit 5

Note that any SQL list is enclosed in parentheses while one in the specification is enclosed in brackets. That is a subtle nuance across the SQL and JSON syntax standards.

Vector Searching Options

There are several ways to specify the vector value. The following are equivalent when using the SQL oriented interaction.

vector = [! listWithSequence(0.1,0.1, 32) !] vector = (0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0) nearVector(0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0)

SQL Between Support

The SQL “between x and y” clause can also be used. For example “price between 10 and 25” is translated into the filter

price: {
   $gte: 10,
   $lte: 25
}

Vectorize $in Support

Here is an example of a SQL query which ends up using the Vectorize $in operator.

select *
from "product-index" 
where name in ( 'Mouse' ) and 
nearText('mouse', null, '@cf/baai/bge-small-en-v1.5')

The low level query specification obtained by pressing Alt-run is shown below.

  

Minimum Score Filtering

Here is an example of using a minimum score as part of the SQL query.

select *
from "product-index" 
where  score >= 0.75 and 
nearText('mouse', null, '@cf/baai/bge-small-en-v1.5')

The SQL clause must be of the form ‘score >= number’. It maps to the minimumScore field of the query specification.

Combining Query Styles

The JSON structure and SQL oriented filtering styles can be combined.

JSON Field	Description
sql	The SQL statement can affect most of the options listed above.
sqlWhere	The string can affect the nearText, nearTextModel, filter and minimumScore options.

For example

{
index: "product-index" ,
sqlWhere: "nearText('mouse', null, '@cf/baai/bge-small-en-v1.5') and price >= 25",
topK: 10,
returnMetadata: 'all'
}

and

{
sql: "select * from 'product-index' WHERE nearText('mouse', null, '@cf/baai/bge-small-en-v1.5') and price >= 25",
topK: 10
}

are both equivalent to

{
  index: 'product-index',
  nearText: 'mouse',
  nearTextModel: '@cf/baai/bge-small-en-v1.5',
  returnMetadata: 'all',
  filter: { price: { '$gte': 25 } },
  topK: 10
}

Using sqlWhere: provides a lot of flexibility. If you would like row properties returned then use the returnMetadata argument. For example

{
index: "product-index" ,
returnMetadata: 'indexed',
sqlWhere: "nearText('mouse', null, '@cf/baai/bge-small-en-v1.5') and price < 25",
topK: 10
}

Troubleshooting

Overview

Vectorize errors are generally propagated back to the user as-is for review. It can be beneficial to review the low level content sent to Vectrorize to better understand these messages. In the Data Source Designer these details can be obtained by pressing ALT and clicking   . There are two other options described below. Any “explain SELECT” or “explain: true” takes precedence over the ALT-click interaction.

SQL

You can enter criteria of the form “EXPLAIN SELECT ….” to have the SQL statement processed and have the returned answer set be the underlying query specification.

A convenient way of specifying this is to have “explain” on the first line and the rest of your SQL on the next lines.

explain
select *
from "product-index" 
where nearText('mouse', null, '@cf/baai/bge-small-en-v1.5')
and price < 25
limit 5

Shown below is the single answer set row.

  

Then simply “comment out” the first line when not in use

// explain
select *
from "product-index" 
where nearText('mouse', null, '@cf/baai/bge-small-en-v1.5')
and price < 25
limit 5

Query Specification

You can also use “explain: true” in the JSON query specification for similar information.

{
explain: true,
returnMetadata: 'all',
index: "product-index" ,
sqlWhere: "nearText('mouse', null, '@cf/baai/bge-small-en-v1.5') and price < 25",
topK: 5
}

Shown below is the single answer set row.

  

Virtual Queries

Qarbine supports several virtual queries as listed in the table below.

Function	Description
List Indexes	List the indexes in the corresponding Cloudflare Vectorize database.
Describe <INDEX>	Describe the details of the given Vectorize index.

See the “DBA Productivity” section for a document describing their uses in Vectorize.

Query by Example Interactions

Below is an example snippet for the product-index content.

  

Qarbine adds the logical _nearText and _nearVector fields for use within the QBE and RBE tools. This allows the user to specify criteria that flows into the automatic query generation process.

For the _nearText property, specify criteria of the form

~'modelName' 'phrase'

For example,

~'@cf/baai/bge-small-en-v1.5' 'Mouse'

generates the SQL

SELECT score,id FROM "product-index" 
WHERE nearText("Mouse", null, "@cf/baai/bge-small-en-v1.5")

This is later translated into the query specification of

{
nearText: "Mouse",
nearTextModel: "@cf/baai/bge-small-en-v1.5"
…
}

The “values” and “vector” fields do not support any criteria. However, either of their selection does result in the answer set including the row embedding.

Cloudflare has very limited support for indicating which metadata fields to return. The options are either “all”, “indexed”, or “none”. As a result, the generated SQL SELECT columns may not correspond to the actual answer set properties. In the Data Source Designer you can use Qarbine pragma “#pragma deleteFields CSV” to define which metadata fields to remove.